Trying to resolve incidents using email

#1

Hi,
I’m using OpenNMS to catch snmp traps from remote devices.
When it receives a trap, OpenNMS matches it to a notification that sends an email to Pagerduty.
The subject of the email is a unique combination of the device type, the remote hostname and trap name.
I also have a second notification that matches the ‘Cleared’ trap.
This sends an email to Pagerduty with the same subject and the email body contains the string ‘Severity: Cleared’ as shown below

image
image.png734×196 30.6 KB

For some reason the incident is not being resolved.

I hope that someone can help me to work out why it’s not working.

#2

Hello @michael.kelly!

Thanks for reaching out. In this case the plugin is provided by OpenNMS and not built by PagerDuty. I would reach out to them either on their Discourse forum - default for community support - or directly on the GitHub project where the code is maintained.

#3

Hi @tiago.barbosa,
Thank you for the quick response.
I’m not using the OpenNMS plugin.
When I added the OpenNMS integration, Pagerduty created a service specific email address.
I have a Notification in OpenNMS that sends an email to that address and it works.
Incidents are created with the information in the email.
When I send an email to resolve an incident, gthe incident is not resolved.
I suspect that this is due to the ‘resolve’ email not being formatted correctly.
I can see that no new incident gets created which would imply that Pagerduty has matched the email to an existing incident.
Are there any examples that demonstrate resolving an incident using email?

#4

I have discovered that I can reproduce this issue using the Email integration with the same custom settings so, nothing to do with OpenNMS.
I guess that I’m doing something wrong but I don’t know what.

#5

Got it. Sorry for the confusion.

You should be able to do it using these instructions.

#6

Hi @tiago.barbosa,
thank you for the response.
The example in the instructions uses something in the subject to trigger/resolve the incident and something in the body to create the alert key and that works.
I’m trying to do the opposite, use the entire subject as the alert key and use a string in the body to resolve the incident but I can’t get it to work.
Is this a supported configuration?

#7

I just tested the integration - successfully - with the configurations you sent on the first message…with one slight change that totally makes the difference. I moved the resolve rule to first place to take precedence over the trigger rule. Since you are using and matching the subject as the key in both rules the first one would always match. Which means that the email would be always matched with the trigger condition and get associated with the first alert, instead of resolving it.

If you invert the order we check if the subject is a match and the body matches the pattern. If it doesn’t then you consider it a trigger. See image below.

image
image.png2574×1478 230 KB

Let me know if that solves the issue for you.

#8

Hi @tiago.barbosa,
when you explain it like that it makes perfect sense.
It works now.
Thank you for your help.

1 Like